Two Sample t-test
data: x and y
t = 0.9, df = 58, p-value = 0.4
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.30 3.35
sample estimates:
mean of x mean of y
10.2 9.2
Cognitive Science Arena 13 - 16 October 2025 | Brixen
Two Sample t-test
data: x and y
t = 0.9, df = 58, p-value = 0.4
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.30 3.35
sample estimates:
mean of x mean of y
10.2 9.2

Welch Two Sample t-test
data: x and y
t = 0.9, df = 35, p-value = 0.4
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.34 3.39
sample estimates:
mean of x mean of y
10.2 9.2
The overlapping (\(\eta\)) is an index of effect size and varies from 0 and 1, formally defined in the following way:
\[\begin{eqnarray} \eta (A,B) = \int_{\mathbb{R}^n} min [f_A (x),f_B (x)] dx \end{eqnarray}\]\(\eta (A,B)\) is normalized to one and when the distributions of A and B do not have points in common, meaning that \(f_A (x)\) and \(f_B (x)\) are disjoint, \(\eta (A,B) = 0\)
where \(B\) is the number of random permutations, \(t\) is the \(t\)-statistic computed on the observed data, \(t_b\) are those computed on the permuted data.
Usually \(H_0\) is defined as the absence of an effect.
As \(\eta\) is the area of overlap, the higher the overlap, the closer the value to 1.
For simplicity, instead of working with \(\eta\) now we will work with the complement \(\zeta\), defined as \(1 - \eta = \zeta\) and define \(H_0 : \zeta = 0\) meaning that there is no difference between the densities of the two populations.
The permutation test is applied to \(\zeta\):
\[\begin{eqnarray*} p = \frac{(\# \hat{\zeta}_b \geq \hat{\zeta})+1 }{B+1} \end{eqnarray*}\]The principle is the same, we shuffle the data and calculate \(\zeta\) B times to calculate the \(p\)-value.
Panel [A] shows the two distributions with \(\eta = .46\) and \(\zeta = .54\), panel [B] shows the distribution of the \(\zeta\) statistic obtained through permutation (\(p < .001\))
We simulated from the Skew-Normal distribution.
For each of the 4 \(\times\) 3 \(\times\) 3 \(\times\) 5 \(= 180\) conditions we generated 3000 sets of data on which we performed the analysis.
For each combination \(\delta\) \(\times\) \(\sigma\) \(\times\) \(\alpha\) \(\times\) \(n\), on the generated data were performed the following tests:
The \(\zeta\) overlapping test is the most powerful choice when assumptions are not met, also with small samples.
As a non-parametric test, it allows to make inference on the entire distribution without making any assumptions.
It is an intuitive index and it allows to investigate group differences beyond the mean .
Be aware of parametric tests assumptions
Always plot your data BEFORE running the analysis
Thinking Before Testing (cit. Richard McElreath )
Here you can find our pre-print, on the implementation of the permutation test for the overlapping index:

ambra.perugini@phd.unipd.it